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LETTER 


from the Foundation 


elcome 2021! 
Like many of you, the Foundation team 

was pretty happy to see that clock strike 
Midnight on January 1, 2021. Last year will definitely 
go down as one of the most memorable in the 
Foundation’s 20-year history. However, we're happy 
to report it ended on a positive note. Thanks to your 
generous support, the Foundation was able to raise 
enough funds to not only continue our efforts to 
support the FreeBSD Project, but also expand our 
software development and advocacy efforts in 2021. 
From new hires in the software development group 
to increased online content and greater focus on 
FreeBSD in education, we're looking forward to all 
of the ways we can continue to help ensure FreeBSD 
is the high-performing, stable and secure operating 
system you've come to rely on. 

Speaking of relying on FreeBSD, this issue of the 
Journal focuses on FreeBSD Case Studies. There’s 
no better way to showcase the benefits of FreeBSD, 
than by hearing it directly from those using it to 
great success. Take a minute to see why companies 
have made FreeBSD their OS of choice, and be sure 
to share the always-free issue with your colleagues 
and friends. 

Thank you again to readers for your continued 
support of the FreeBSD Project and Foundation. We 
wish you a very Happy New Year and look forward 
to all we are going to accomplish together in 2021. 


On behalf of the FreeBSD Foundation, 
Deb Goodkin 
FreeBSD Foundation Executive Director 
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WeGet!etters 


Oh Generous, Grandiloquent, Gratuitous 
FreeBSD Journal Letters-Answerer: 





FreeBSD 13 comes out any time now. It has a whole 
bunch of features I’m eager to get my hands on, but 
I’m leery of a brand-new release. Any advice on 
when I should upgrade? 


And do they really pay you in gelato? 


Thanks, 
—Doesn’t Need Outages 


DNO, 

| know this story. 

Early in your career you fell under the tutelage of a grizzled sysadmin, the sort who lost an 
eye in the Unix Wars, detonated a lobe of his liver bootstrapping the K&R C compiler, and kept 
a copy of the Alpha boot loader encoded in the knots of his flowing gray beard. You asked him 
this same question about some other new release. He picked up the Free the Berkeley 4.4 cof- 
fee mug where he kept the knucklebones of the last Ultrix salesman who dared radiate body 
heat in his artisanally cooled datacenter, gave it a good shake, and cast the bones across his 
desk to read the wisdom therein. Between the roar of the racked servers all around and the way 
he'd wrecked his vocal cords screaming at the University of Minnesota's Gopher developers over 
changing their server to a paid license, you had to listen carefully to sieve his hoarsely whispered 
wisdom from the noise. 

“Never, never install a .0 release.” 

That’s the sort of thing that makes quite an impression on a young sysadmin. 

In Old One-Eye’s defense, that was unsurpassed wisdom in the Dialup Age. The fastest 
way to download an operating system release was to get a backup tape by mail order. Criti- 
cal patches were distributed by Usenet, if you were lucky enough to have an account on a site 
with a mighty 1.544-megabit uplink. Inexpensive servers cost several thousand dollars, or a 
transplant-ready liver if you could find an insufficiently cautious salesman who hadn't already 
wrecked theirs. 

It's a different world now. For one thing, we have salespeople. And they've all been warned 
about the liver thing. 

If you're eyeing a .0 release today—you're already too late. 

Modern operating systems are public, exactly like a sleazy Hollywood star's collection of inti- 
mate infections. The time to find problems is before the release. | don’t care what flavor of Unix 
you run, they're public. Even closed-source Unix developers give their customers access to pre-re- 
lease media, though I’m certain | don’t know why you'd want to grant them more help than 
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your outrageous license fees. The developers have asked, begged, cajoled, pleaded, and threat- 
ened their user base to test release candidates, in-progress versions, snapshots, and patches for 
months or years. And here you are, asking if you can trust the finished product? 

You selfish dweeb. 

Grab the most recent snapshot, release candidate, or whatever's the latest and greatest, and 
try it in your environment. Configure it with all the debugging and prepare for kernel dumps. 
Test your applications under load. Tell the developers what worked and what doesn't. 

If you're reading this right after 13.0 came out and are all sorts of relieved that you don’t have 
to do this work, guess what? A pre-pre-release 14.0 is available this very moment! Or maybe all 
the good topics for PhD theses in Irrelevant American Authors have been taken, but despera- 
tion has driven you to delve into the moribund, unrecognizable text archives of a long-telepathic 
Journal to identify the moment when my descent into ferality crossed into forthright malignance, 
and release Ax67.str is now in development. Whatever the case, there's a forthcoming release 
available for testing. 

No, l'm not saying that you should deploy the release candidates and development ver- 
sions on every host across your environment. People should, but anyone asking this question 
shouldn't. You're not equipped. 

Testing development releases requires not only sysadmin skills, but sysadmin practices. What's 
the difference? Skill means you know how to do the things you should do. Practice means you 
perform those things. You need backups. You need to know that those backups can be re- 
stored. You must not only know how to submit bug reports, you need to be comfortable sub- 
mitting them. The whole point of running one of these early versions is to report on bugs. 

For your own sanity, you need to deploy development versions intelligently. 

Don't slam development releases onto every web server in the cluster. Pick one or two. Put 
them in the load balancing pool. See what happens. Compare responsiveness under similar 
loads. Configure them to automatically dump and reboot in case they panic. If they're running 
ZFS, keep known, good boot environments on hand. While “the kernel panics every one thou- 
sand sixty-three seconds, here’s the text dump” is eminently valuable, there’s no need to live 
with that once you've identified the problem. 

If you keep working it, you'll eventually learn that you can run development versions every- 
where. People do. You can become good enough to join them. 

Yes, this requires allocating time and hardware. That's cheap. If you doubt me, go price suffi- 
cient Oracle Solaris licenses and servers to host your environment. In a big enough company, you 
can hire official testers and still save enough to feed the staff pizza and beer every day for lunch. 
Just be sure you give the Oracle rep a burner phone number and an email address in a burner 
domain name, because like the Terminator, they will never stop, never show mercy or pity, and 
never tire until they own your scrawny, underfed soul. 

You know what improves your soul? What makes your soul blossom? What develops great- 
ness of character and mind? What sharpens your sysadmin chops until none can stand against 
you? 

Testing development versions of the software you depend on. 

Deploying them. Using them day-to-day. Providing feedback for the developers. Bug reports 
are great, but so is “I’m running the latest in production to serve four and a half trajillion Ham- 
sterSoft queries a second, and it’s going great.” Negative results are still results. 

You know what else improves your soul? 
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Paying your Letters Column guy his gelato. | had expected George to settle up at BSDCan in 
spring 2020, but he never showed up. Maybe he'll be there in 2021. If he doesn’t come through, 
though, I'll have to write it off as a bad debt. 

These columns might get a little cranky if that happens. Consider yourself warned. 








Have a question for Michael? 
Send it to letters@freebsdjournal.org 











MICHAEL W LUCAS's most recent books include SNMP Mastery, Cash Flow for Creators, 
and Drinking Heavy Water, plus a bunch more at https://mwl.io. Under no circumstances is he 


allowed near users. 





eo FreeBSD 


The FreeBSD Project is looking for 


* Programmers ° Testers 





* Researchers * Tech writers 


* Anyone who wants to get involved 


Find out more by 


Checking out our website 
freebsd.org/projects/newbies.html| 


Downloading the Software 
freebsd.org/where.htm| 


We're a welcoming community looking 
for people like you to help continue 


Join us! 


Already involved? 





Don't forget to check out the latest 
grant opportunities at 
freebsdfoundation.org 


developing this robust operating system. 


Help Create the Future. 
Join the FreeBSD Project! 


FreeBSD is internationally recognized as an innovative 
leader in providing a high-performance, secure, and stable 
operating system. 


Not only is FreeBSD easy to install, but it runs a huge number 
of applications, offers powerful solutions, and cutting edge 
features. The best part? It’s FREE of charge and comes with 
full source code. 


Did you know that working with a mature, open source 
project is an excellent way to gain new skills, network 

with other professionals, and differentiate yourself in a 
competitive job market? Don't miss this opportunity to work 
with a diverse and committed community bringing about a 
better world powered by FreeBSD. 


The FreeBSD Community is proudly supported by 


FBSD 


FOUNDATION 
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of FreeBSD BY DRU LAVIGNE 








This column aims to shine a spotlight on contributors who recently received 
their commit bit and to introduce them to the FreeBSD community. 

In this installment, the spotlight is on Juraj Lutter, who received his 
ports bit in December 2020. 


Tell us a bit about yourself, your background, and your interests. 


e Lutter: | was born in the former Czechoslovakia. My father is an electronics enthusi- 
ast and bought our first home computer (Sinclair 2X81), which was soon replaced by a 
more modern successor (ZX Spectrum). So as a boy, | got to know BASIC and later also 
the Z80 assembler. After the social changes that took place in our country in 1989, 16-bit 
computers, which were not available to private individuals until then, became more wide- 
spread in our country. This allowed me to get acquainted with DOS, PC hardware, Turbo 
Pascal, Turbo C, Turbo Assembler, and other tools. 

| have been in the IT industry since the late 1990s. First as a PC service technician, then 
as a system administrator at an ISP and as a freelancer, | have been working in the system inte- 
gration industry. In addition to FreeBSD, where | maintain various ports, | am also interested in 
other open source technologies such as SmartOS and illumos, various infrastructure programs 
(BIND, PowerDNS, Zabbix), databases (PostgreSQL), computer networks (switching, routing, 
firewalling), and data storage (either monolithic storage or ZFS). | also have a commit bit to 
okgsrc for SmartOS and NetBSD. 

| collect old computers (especially 8-bit computers from Sinclair) and also devote some of my 
time to electrical engineering and electronics. 

| live in our capital, Bratislava, and have two children—an 8-year-old son who is starting to 
discover Python and a 10-year-old daughter who is more sympathetic to the LUA language. 


Ce ee RRR a 


How did you first learn about FreeBSD and what about FreeBSD interested you? 


e Lutter: | first encountered UNIX around 1995, specifically SCO Unix 3.2, where one of my first 
tasks was to configure UUCP over an X.25 line. Shortly afterward, | received a 4-CD set from 
Walnut Creek CD-ROM that included among other things one of the first releases of 

Slackware Linux. And since | was already in contact with UNIX (SCO), | was interested in 

Linux because it was easy to install and use when compared to SCO UNIX. In my fourth 

year of high school in the fall of 1996, a classmate mentioned FreeBSD to me, and short- 

ly thereafter | received my first user account on a FreeBSD server, which | used mainly for 
e-mail. Subsequently, around 1999, | started working as a system administrator in the Slo- 

vak branch of Nextra (Telenor Internet), where FreeBSD was deployed on several dozen 
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servers on 1386 and Digital Aloha platforms and where we also had a Slovak FreeBSD mirror 
(www, ftp, CVSup). 

FreeBSD fascinated me because, compared to Linux, it is a compact system, where the ker- 
nel and userland are in harmony and symbiosis, and where ports make it very easy to maintain 
software packages for different servers (for example, using poudriere). | also still maintain our 
local FreeBSD mirror. 


(E E E RRR 


How did you end up becoming a committer? 


e Lutter: The more | used FreeBSD, the more | ran into bugs in individual ports and the base 
OS. Over time (since 2004), | started to open bug reports and contribute patches. It wasn’t un- 
til the end of last year that | spoke to Sergey A. Osokin, to whom | mentioned that sometimes 
the bug reports | open remain unresolved for a long time and that | would also like to contrib- 
ute to the development. Sergei suggested that he ask about it. And one day, | received mail in 
which René Ladan informed me that | had become part of the FreeBSD development commu- 
nity as a ports committer. | was really very pleased and honored. | want to thank Sergey for the 
opportunity join this amazing community and also Steve Wills for his help and answers to my 
curious questions. 


eoeeoeeeveeeee eooeoeeveveeeeee eee eee ee ee ew ew ew 8 eooeoeeeeeeeeee eee ee ee © © oO 8 


How has your experience been since joining the FreeBSD Project? Do you have any 
advice for readers who may be interested in also becoming a FreeBSD committer? 


e Lutter: Since I've been contributing patches for a long time, I’ve seen how committers and 
the commit process works and how code reviews work (everyone who contributes should 
learn phabricator). And that’s why my impressions are still positive. Thanks to constructive dis- 
cussion, sharp edges can always be smoothed out and there are many things to learn from the 
more experienced committers and vice versa. 

It is also a good idea to read (and eventually remember) the information in the Porter's 
Handbook and Committer’s Guide. For example, | found a lot of information there about Sub- 
version and the processes built on it (like MFH and the like). 


DRU LAVIGNE is the author of BSD Hacks and The Best of FreeBSD Basics. 
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Tarsnap’‘s 
FreeBSD Cluster 


BY COLIN PERCIVAL 


Tarsnap is an online backup service with an emphasis on security—indeed, when | started the 
company back in 2006, | quickly settled on the tagline “Online backups for the truly paranoid” 
to reflect the fact that as a cryptographer and FreeBSD Security Officer, my aim was to provide 
a service which was secure enough that | would trust it with my own secrets. Tarsnap is also 
one of the leading reasons that FreeBSD is available in Amazon EC2: Tarsnap needed to run in 
EC2 (among other things, so that it had cheap and fast access to the Amazon S3 storage ser- 
vice), but | needed an operating system which | trusted and 


eTarsna ap knew I could administer easily—in other words, Tarsnap needed 


one backups for the truly paranoid FreeBSD in EC2, and | scratched my itch. 


Customized AMIs 

While all of the EC2 instances Tarsnap uses run FreeBSD, none are ever launched from the 
“stock” FreeBSD Amazon Machine Images (AMIs) which the FreeBSD project publishes. Instead, | 
make use of a tool | created five years ago to build lightly customized FreeBSD images: An “AMI 
Builder” AMI, available for FreeBSD 12.2-RELEASE as ami-085ee41974babfifi in the us-east-1 
EC2 region. These AMI Builders are not currently provided by the FreeBSD project but are instead 
something | build and publish myself after each release; at some point | hope to integrate these 
into the builds performed by the release engineering team. 

To create a “Tarsnap FreeBSD 12.2-RELEASE” image, | start by launching the aforementioned 
AMI Builder—and then | wait, roughly 20 minutes, while the virtual machine spins up and installs 
(stock) FreeBSD 12.2-RELEASE onto its virtual disk. This disk is then mounted on /mnt/ while the 
FreeBSD system mounted at / runs from a memory disk and starts an sshd process. 

Once | can SSH into the AMI Builder (like other FreeBSD images in EC2, using the SSH key | 
provided to EC2 and the user name ec2-user), | set to work with some standard configuration 
which | want on all of Tarsnap’s systems: 

e| build and install some packages from the FreeBSD ports tree: pkg, djbdns, qmail, spiped, 

and tarsnap. 

e| enable some daemons | want (svscan and spiped), disable other code | don’t want (sendmail 

and firstboot pkg installation), and lock down some settings (disabling network listening in 
syslogd and restricting sshd to IPv4) in /etc/rc.conf. 

e| instruct pkg to use packages | build myself instead of the packages distributed by the 

FreeBSD project, by creating configuration files in /usr/local/etc/pkg/repos/. 

e| add cron jobs to run freebsd-update cron and pkg upgrade -qn every morning— 

| don’t want updates installed automatically, but | definitely want to get an email when 
they're available. 

e| set up djbdns to provide a local DNS cache and set resolv.conf to point at it. 
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e| set up qmail to send outgoing email via my mail server. 

e| set up spiped to create secure connections to my mail and package servers, and also to 

wrap incoming SSH connections. 

Finally, after performing all the configuration | want on /mnt/, | unmount the disk and ask EC2 
to “create an AMI from the running EC2 instance”. Despite the description of this EC2 API call, 
the AMI created does not reflect what was running, but rather the current state on disk—in oth- 
er words, it ignores the FreeBSD system running out of a memory disk and creates an AMI corre- 
sponding to the configuration | performed on the filesystem mounted at /mnt/. 

Now | have a configured “Tarsnap FreeBSD” image from which | can launch instances with all 
of my preferred defaults set up, and there’s one last step: | ask EC2 to copy this AMI from the us- 
east-1 region to the us-west-2 region. While almost all of Tarsnap’s servers run in us-east-1 (which 
was the only EC2 region when | launched Tarsnap) | do have one system in us-west-2: A monitor- 
ing system which alerts me if anything breaks. 


pkg Builder 

The FreeBSD Project provides binary packages for software in the ports tree, and most users 
will want to make use of those rather than building from source. For Tarsnap’s servers, | do my 
own builds, for two principal reasons: 

eln some cases, | want non-default port options. 

e As a FreeBSD developer, sometimes | commit updates and want to use them immediately, 

rather than waiting for the next scheduled package set. 

It's possible that at some point in the future neither 
of these will apply—it may be that the ports where | set 
non-default options will gain “flavors” which provide what z ji ii 
| need, and it may be that the FreeBSD Project will someday Tarsnap s “cluster 


perform shockingly fast package builds every time a port is includes two very lightly 
updated (but considering the rate at which new compilers 
are released with ever-worsening performance, this seems used web servers. 


unlikely). In the meantime, running my own package builds is 
easy and convenient. 

| use poudriere for this purpose, and the process of set- 
ting up the builds is very easy: After installing poudriere, simply poudriere ports -c and pou- 
driere jail -c -j JAILNAME -v 12.2-RELEASE. After that, | set a few options in /usr/ 
local/etc/poudriere.d/make.conf, set a cron job to run poudriere bulk -f /root/ 
pkgs-wanted (where | have a list of packages | want), and use lighttpd to serve up the resulting 
packages. 

Building a complete set of the packages | use on Tarsnap’s servers takes about 2 hours on the 
$15/month “t3.small” EC2 instance | use, but most package build runs complete far faster than 
that, thanks to poudriere operating incrementally and not recompiling unchanged packages. In- 
deed, | would use an even smaller EC2 instance but for one detail: Some of the C++ compiles fail 
on instances with only 1 GB of RAM. 


Web Servers 
Tarsnap’s “cluster” includes two very lightly used web servers, running on EC2 “t3.nano” in- 
stances. In an era of web frameworks and rich web applications making javascript function calls, 


the Tarsnap website is perhaps exceptional mainly for its archaic design: The public portion of the 
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website is entirely static HTML—hand-written aside from a shell script which wraps content with 
a header and navigation section—and the account creation and management code consists of a 
handful of CGI scripts... written in C. While CGI scripts have inherent performance problems— 
forking a process is far more expensive than running code within the context of the web server 
or forwarding requests to another long-running daemon—as far as Tarsnap is concerned, | would 
love to be in the position of experiencing performance problems due to an excess of customers 
using the website. 

There are a few slightly more modern aspects of the web servers, however: TLS traffic is un- 
wrapped using hitch, in order to keep the cryptographic code segregated from the web server 
code; TLS certificates are obtained using Let's Encrypt and certbot; and TLS private keys are kept 
in an Amazon Elastic File System (aka. NFS) filesystem. While the use of hitch, Let's Encrypt, and 
certbot are very common, the use of Amazon EFS probably deserves some explanation. 

The use of Amazon EFS solves a bootstrapping problem: | obtain TLS certificates via the “web- 
root” mechanism, wherein a certificate issuing request is authorized by placing files into a /.wel1- 
known/acme-challenge directory on the website. This only works once traffic for the website is 
directed to the server in question—but | don’t want to direct traffic to a new host until it is ready 
to respond to HTTPS requests. Storing private keys in a manner which survives the replacement of 
a web server instance solves this problem. 

Now, NFS is not known for having a stellar track record with regard to security, and operates 
in plaintext, neither of which seems ideal for cryptographic material—but the fact is that the 
guarantees made by TLS are now sufficiently weak that no security is lost here. If an attacker 
can intercept traffic on the Amazon EC2 network, they can trick Let’s Encrypt into issuing them 
a new certificate allowing them to impersonate Tarsnap’s web servers—so having another at- 
tack which would rely on intercepting traffic on the Amazon EC2 network does nothing to ex- 
tend their capabilities. 


Mailserver 

All of Tarsnap’s email is routed through a single mailserver. This includes: 

e”“ Human” emails sent between me and the outside world. 

e “Logging” emails generated by cron jobs (which land in my inbox). 

e Transactional emails generated when users sign up for Tarsnap, confirm their email addresses, 

make payments, or need to be warned that their (prepaid) accounts are running out of money. 

e Public mailing lists, both for Tarsnap announcements and discussions, and for open source 

software which originated from Tarsnap. 

For historical reasons, the Tarsnap mail server runs qmail; specifically, “this is what | started with 
when | configured my first FreeBSD server, nearly 20 years ago.” If | were starting from scratch, | 
would probably use postfix, but in the long-standing tradition of sysadmins everywhere, as long 
as it isn’t broken, I’m unlikely to fix it. 

Email arrives at the mailserver via two routes: Connecting to port 25 on the external network 
interface and connecting to spiped via port 8025—which then arrives at qmail via port 25 on the 
loopback interface. Using spiped in this manner not only secures “internal” network traffic, but 
also neatly solves the question of email relaying: Any email which arrives at qmail via the loopback 
interface can be safely relayed to other domains. 

Since | use qmail, | naturally use Dan Bernstein's ezmlm (and its extension, ezmlm-idx) to man- 
age Tarsnap’s mailing lists. The “tarsnap announcements” list is moderated, but the rest are open 
to postings from any subscriber; fortunately, | have had no problems with trolling, and thus far 
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spambots don’t seem to be smart enough to subscribe to mailing lists before attempting to send 
email through them. 

Tarsnap’s mailing lists have archives accessible via HTTP/HTTPS. | use mhonarc to take the mail- 
ing list posts from ezmlm-idx and convert them into HTML format; lighttpd to serve them up to 
the world; and hitch to add a layer of TLS for those who want it. It’s hard to imagine what secu- 
rity is added by TLS here: The mailing list posts are public, and the number of bytes transferred is 
enough to uniquely identify the page being downloaded; nonetheless, some people strongly pre- 
fer to use TLS even when it serves no purpose. 

Finally, outgoing email is dispatched via a shell script which runs one of two “qmail-remote” 
programs: Either the original qmail-remote, which sends email directly via SMTP, or a “qmail-re- 
mote-ses” | wrote which sends email via Amazon's Simple Email Service. Unfortunately delivering 
email into inboxes is increasingly difficult, so for transactional emails | hand the job over to Ama- 
zon; at a cost of $0.10 per 1000 outgoing emails, Amazon doesn’t need to be very much better 
at delivering email to pay for itself when it comes to customers signing up to give me money. On 
the other hand, people who have overly restrictive mail servers are unlikely to subscribe to Tars- 
nap’s mailing lists in the first place—so | have no problem with relying on qmail and direct SMTP 
for that email traffic. 


Monitoring 

As mentioned earlier, while most of Tarsnap’s systems are in Amazon’s us-east-1 region, | have 
a monitoring system in the us-west-2 region. There are two reasons for having this instance in a 
different region from everything it is monitoring: First, to al- 
low it to detect outages related to the AWS region's external 
network connectivity; and second, to minimize the likelihood 
that a failure disables both Tarsnap’s systems and the moni- 
toring simultaneously. Furthermore, even if an outage does of simple shell scripts 
knock two AWS regions offline simultaneously, it’s (a) unlike- 
ly that there’s anything | could do to respond to it, and (b) to perform a range 
very likely that the world is facing far more important prob- TEN 
lems than Tarsnap being offline. of monitoring tasks. 

Rather than using any of the widely used monitoring 
frameworks, | use a handful of simple shell scripts to per- 
form a range of monitoring tasks (pinging servers, fetching web pages, performing Tarsnap back- 
ups) from cron jobs; these each emit a state (“GOOD", “FAILED”, or “TIMED OUT”), and another 
script uses Twilio to send SMS messages and make phone calls. The combination of a small num- 
ber of notifications and Twilio’s usage-based pricing makes this extremely affordable—indeed, 
most of the cost is the $1/month fee to rent a phone number, which is required in order to send 
SMS messages. 


| use a handful 


Holding Everything Together: spiped 

The astute reader will have noticed that I’ve mentioned spiped a few times but without giving 
many details. Since this is a little-used tool—and one | wrote myself specifically for securing con- 
nections to and between Tarsnap systems—l think it deserves special attention. 

The spiped utility, at its core, is a tool for connecting one socket address to another socket ad- 
dress in a cryptographically secure manner. In this sense it can be seen as a replacement for stun- 
nel or “ssh -L” port forwarding; but whereas stunnel relies on the overly complex TLS protocol and 
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certificates, and ssh uses a persistent TCP connection over which tunneled connections are multi- 

plexed, spiped uses a pre-shared secret and opens a new connection for each connection being re- 
layed—making it much simpler and giving it the same behavior as TCP with regard to network fail- 
ures. At present, spiped supports TCP sockets over IPv4 and IPv6, as well as local (“UNIX”) sockets. 

The name's origin, “secure pipe daemon,” may give another hint to the intent, however: 
Whereas the introduction of pipes to UNIX in 1973 led to an explosion of functionality as utilities 
were combined in various ways, | wanted to develop software which used a multitude of dae- 
mons—and to be able to connect them without regard for whether they would end up running 
on the same host or across insecure networks. 

As mentioned above, | use spiped to secure SMTP connections to Tarsnap’s mailserver; | also 
use it to secure POP3 connections, which has the benefit of allowing me to use POP3 in its sim- 
plest and least secure mode—no TLS and plaintext passwords—while retaining the necessary se- 
curity. (Hey, it works, ok? I've been meaning to switch over to IMAP for decades.) 

Similarly, connections to the pkg builder are protected by spiped; while the contents of the 
packages are not sensitive (they all come directly from the FreeBSD ports tree) this allows me to 
ensure package integrity without needing the more heavy-weight option of using poudriere to 
sign packages and may also protect the pkg builder somewhat in the (unlikely) event that a vul- 
nerability is found in lighttpd. 

Finally, | use spiped to protect all of my SSH connections—while all of the systems | use 
have sshd enabled in order to allow me to easily administer them, port TCP/22 is blocked and 
the only way to reach sshd is via an spiped process which doesn’t allow any traffic through 
until the incoming connection has been cryptographically authenticated. 





Legacy Systems 

Of course, in addition to the aforementioned infrastruc 
ture, there’s the Tarsnap backup service itself. This runs on Over time | expect to 
what one might call “legacy” systems: Since it is directly re- 
sponsible for ensuring the safety of customer data—not use more open-source 
to mention bringing in revenue—l allow the backup ser- 
vice to lag behind other systems (except where security is code in the backup 
concerned, naturally). At present it uses an older version of Service. 

FreeBSD on an older EC2 instance type—a fact which pro- 

tected it from the stability problems in the (recently added) 

ENA network interface driver which were corrected in the 

FreeBSD-EN-20:11.ena Errata Notice—and the only third-party packages installed are those which 
| have in my “Tarsnap FreeBSD AMI”—i.e. those needed to allow me to securely connect for ad- 
ministration purposes, and those used to send outgoing email. Aside from those, the only code 
running is Tarsnap’s proprietary code—a few daemons, plus cron jobs for internal performance 
monitoring and nightly billing purposes. 

Over time | expect to use more open-source code in the backup service—or perhaps | should 
say, | have released open source code which | expect to be using in the backup service in the fu- 
ture. | designed the kivaloo data store around Tarsnap’s needs for metadata storage, and it only 
made sense to release the code first and use it later. Tarsnap stores customer data in Amazon 
S3, but it’s necessary to keep track of where each data block was stored in order to be able to 
retrieve it on demand. This leads to a data store usage pattern—tables with keys and values of 
roughly 40 bytes each—for which most data stores are poorly optimized. 
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Future Directions 

| can’t conclude without mentioning something Tarsnap isn’t using yet, but which I'd like to 
use: “Graviton 2” arm64 EC2 instances. In all of my testing to date these have performed ex- 
tremely well—and they're roughly 40% less expensive than the x86 EC2 instances currently used 
by Tarsnap. 

Naturally, while FreeBSD images are available for arm64 EC2 instances, Tarsnap won't be mak- 
ing use of them (aside from development and testing) until the arm64 architecture is fully sup- 
ported by the FreeBSD Project—including having binary updates available via FreeBSD Update. 
| understand that this is likely to arrive in time for FreeBSD 13.0; so, by this time next year | may 
have replaced many systems with new arm64 instances. (The core backup service, of course, will 
continue to run on x86 for a while longer.) Only time will tell, but it’s safe to say that this is an 
area where I’m very excited about the future. 


COLIN PERCIVAL has been a FreeBSD developer since 2004 and was the project’s Security Of- 
ficer from 2005 to 2012. In 2006, he founded the Tarsnap online backup service, which he con- 
tinues to run. In 2019, in recognition of his work bringing FreeBSD to EC2, he was named an 
Amazon Web Services Hero. 
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BALLY WULFF 


BY MATEUSZ PIOTROWSKI 


BALLY WULFF Games & Entertainment GmbH is a prominent German company in the enter- 
tainment electronics segment that develops, produces and sells cash gaming machines. With its 
headquarters located in Berlin, BALLY WULFF operates not only in Germany, but also in Spain, 
and currently employs around 300 people. 

Since the early 2000s, FreeBSD has been the platform of choice for BALLY WULFF prod- 
R WW ucts. Thanks to its invaluable stability and consistency, the system 

engineering team is able to meet the ever-changing market de- 
ally toona mand whether that is for small, disk-space footprint, higher secu- 
rity measures, or better graphics. Oftentimes, the team contributes code and documentation 
patches back to the community. In addition, many BALLY WULFF employees have served as 
FreeBSD committers. 

BALLY WULFF is well known for its development process happening entirely in-house. The 
final products are the collective work of various BALLY WULFF teams. Collaboration among 
product designers, hardware engineers and game developers is the name of the game. Every- 
thing from the design of the machines, hardware integra- 
tion and game development, to the final production and 
assembly of the machines takes place in-house. FreeBSD is FreeBSD is at the heart 
at the heart of it all not only as the operating system in the 
final gaming machine, but also in the development work- of it all not only as 
stations and production appliances. 

This short case study provides a deeper insight into the operating system 
BALLY WULFF system engineering team’s goals and in the final gaming 
explains how FreeBSD helps to achieve them. 





machine, but also 
Goal 1: Limiting the Disk Footprint of the OS 
A BALLY WULFF gaming machine can be thought of in the development 
as a huge video game console. What may come as a sur- workstations and 
prise in the era of loT is that it is not connected to the In- 
ternet, but instead ships with all the software, games and production appliances. 
their assets preinstalled. Once it leaves the production site, 
the only way to change the software on a machine is via a 


manual update procedure that involves physical storage media like USB sticks. Although no lon- 


ger a pressing issue, the older generations of gaming machines posed an interesting challenge 
for the system engineers. Disk quotas for each team had to be carefully balanced so that every 
game and administrative tool received its fair share of a disk. However, the disks always turned 
out to be just a little bit too small to fit all the desired data. As a result, the operating system 
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had to be stripped of all unnecessary bits. FreeBSD, like other, well-designed software projects, 
provides a great number of build options capable of excluding everything but essentials from 
being compiled. 

Unfortunately, the standard build options were not enough. It turned out that in order to 
achieve the desired disk footprint, the system engineering team had to gain control of a great- 
er granularity over the build process. Luckily, the FreeBSD build system has been engineered, 
maintained, and constantly improved with customizability and stability in mind. It is by design 
that downstream consumers (and appliance vendors in particular) are able to look under the 
hood of the build system, modify it as needed, and expect only a minimal maintenance over- 
head caused by the local changes to the source tree. 

BALLY WULFF has maintained an internal patch set for the FreeBSD build system to exclude 
non-essential files from final OS images. This appliance-specific patch set has naturally fit into 
the build infrastructure and does not feel like an external add-on bolted on to an existing en- 
vironment. At the same time, it has not caused a significant maintenance burden to the sys- 
tem engineering team. As a result, the disk footprint of the OS has been limited to a minimum, 
leaving more disk space for games—a real value to customers. 


Goal 2: Shipping Modified Packages 

The FreeBSD Ports Collection has proven to be an invaluable asset to BALLY WULFF over the 
years. It offers a standardized and expandable way of customizing and adding additional soft- 
ware to the OS. In fact, it is so straightforward that creating customized packages is one of the 
first things new FreeBSD users learn about. 

FreeBSD ports developers make sure that the ports 
framework evolves steadily and stays backward-compatible 
for many years. So even though the FreeBSD Project has It is easy for the 
already switched to poudriere within its packaging infra- 
structure, users like BALLY WULFF can still migrate at their system engineering 


convenience. Ultimately, FreeBSD is all about stability with 
no unpleasant surprises. As a result, it is easy for the system team at BALLY WULFF 


engineering team at BALLY WULFF to keep up with the to keep up with 
changes and plan ahead. 
BALLY WULFF maintains—internally—a fork of the the changes 


FreeBSD Ports Collection with a handful of additional com- 
pany-specific ports and patches for the existing ports. Not 
only is backporting of the latest versions of ports incredi- 
bly easy, but maintaining a custom version of an existing 
port is also simple and painless. Another advantage of the 
FreeBSD Ports Collection is that the distribution of packages via an internal package repository 
is effortless and well-supported. 

The FreeBSD Project is constantly adding new improvements to ease the process of extend- 
ing private collections of ports that benefit BALLY WULFF directly. The latest example is poudri- 
ere, which streamlines package building, testing, and publishing processes. Another important 
feature is an overlay support for ports trees, which is currently being tested by the system engi- 
neering team at BALLY WULFF. There is a high chance that it will alleviate the need to keep an 
internal fork of the ports tree, further reducing maintenance overhead. 


and plan ahead. 
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Goal 3: Customizing System Startup 

Typically, general-purpose operating systems feature a program to control the system start- 
up. It usually configures the newly booted system, for example, by mounting disks and starting 
essential system services like networking. 

In the case of a BALLY WULFF gaming machine, the system start-up procedure is quite dif- 
ferent from a typical desktop. The standard FreeBSD start-up procedure is configurable enough 
to cover most use cases for servers and desktops, but in the case of a gaming machine, it 
made sense to replace the standard rc(8) mechanism completely. Luckily, there is no black 
magic involved in replacing the standard rc(8) framework with a custom one. Actually, replac 
ing the /etc/rc file is enough to get started. As a result, the system engineering team at BALLY 
WULFF maintains a dedicated system start-up script that prepares the OS environment for the 
games to launch. 

At BALLY WULFF, the customized rc(8) framework has been in use for many releases and it 
continues to work flawlessly. This is certainly a benefit of FreeBSD’s steady development prac 
tices and modularization of the base system—it is absolutely reasonable to customize a part 
of FreeBSD and expect the rest of the system to work just fine. It definitely gives the develop- 
ers peace of mind and lets them focus on developing what is important rather than constantly 
catching up with backwards-incompatible, upstream changes. 


Goal 4: Supporting Custom Update Procedures 

It should come as no surprise that BALLY WULFF gaming machines require regular updates. 
Platform updates occur when there is a need to squash an annoying bug or add an important 
business functionality to an already-released and operating machine. Much more often, howev- 
er, the machines are updated with new games. The update process of the gaming machines is 
one of the most important and rigorously tested procedures in the company. Thorough testing 
and QA checks guarantee that the updates applied to the machines already operating in the 
market are not going to cause any unnecessary downtime. The update process must allow for 
unsupervised and automatic installation of new software. Rendering the machine inoperable in 
the course of an update is out of the question. 

Due to the architecture of the gaming machine's operating system, it would be unnecessar- 
ily complicated to update the system with freebsd-update(8) and pkg(8). Thankfully, FreeBSD's 
simplicity allows for implementing a completely custom update procedure. 


Goal 5: Running the Same OS in Both Production and Development 

One of the golden rules of software engineering is that develooment should happen in an 
environment identical or at least closely resembling the production environment. Developers do 
not have to debug their code twice when working in a unified environment. 

BALLY WULFF game developers use FreeBSD workstations to test games before trying them 
out on the actual gaming machines. Amazingly, FreeBSD powers the gaming machines and the 
workstations equally well. 

It is worth noting that the game development department is many times larger than the 
FreeBSD team at BALLY WULFF. Nevertheless, maintaining an internal distribution of FreeBSD 
tailored specifically to the game developers’ needs is very doable. The same FreeBSD-based OS 
runs on both a gaming machine and on a development workstation, the difference being main- 
ly the list of installed packages. This is a great benefit of FreeBSD being a general-purpose OS. 
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Goal 6: Staying Close to the Community 

Being close to the community allows BALLY WULFF to both participate in the development 
of FreeBSD and to stay in contact with FreeBSD developers. For example, BALLY WULFF aims to 
keep the amount of local FreeBSD patches to a minimum. The maintenance burden of non-es- 
sential patches is simply not supportable. Not only is upstreaming patches a very sound busi- 
ness decision due to additional testing, but it is also a great way to give back to the FreeBSD 
community. Most of the time, however, the FreeBSD patches developed internally at BALLY 
WULFF are too vendor-specific and not suitable for inclusion in the FreeBSD source trees. Nev- 
ertheless, the company makes sure to contribute in other ways as well. BALLY WULFF devel- 
opers regularly participate in FreeBSD Developer Summits and open-source conferences like 
FOSDEM and EuroBSDcon. In 2019, BALLY WULFF hosted a DevSummit organized at the com- 
pany’s headquarters in Berlin. 


Summary 

FreeBSD has been a great OS for BALLY WULFF thanks to its remarkable build system, which 
has been developed and maintained in a way that allows for effective adaptation of the sys- 
tem to specialized appliances. In the past, a major benefit of FreeBSD to BALLY WULFF was the 
small, yet functional, base system that could be stripped down even further by utilizing existing 
build(7) knobs or by introducing vendor-specific changes to precisely control what is included in 
the final OS image. The internal patches to FreeBSD base and ports build systems fit naturally 
into the consistent Makefile-based infrastructure. All those features allowed the BALLY WULFF 
system engineering team to minimize the size of the OS, which, in turn, left more space for 
games and their assets as the BALLY WULFF developers continued to push hardware and soft- 
ware limits. 

Now that disk space is not as precious as it once was, the need for a special patch set re- 
ducing the final size of the OS is gone. The focus and energy at BALLY WULFF are directed to- 
ward other aspects of system development. It is no longer necessary to heavily modify FreeBSD 
sources to optimize for small disk footprint. The transition to building the OS from the unmod- 
ified FreeBSD sources is underway. So far, it has been painless and beneficial as it significantly 
simplified the build infrastructure. This is a result of a herculean effort by the FreeBSD commu- 
nity to maintain backward compatibility wherever feasible. Every major change to the FreeBSD 
system is implemented with downstream consumers workflows in mind. 

The computing world has evolved quickly and the team’s focus is no longer on making the 
system footprint as small as possible. The target now is to increase the robustness of the system 
and to keep maintenance costs low. Ultimately, the goal of the BALLY WULFF system engineer- 
ing team is to provide game developers with a performant and stable gaming platform. 


MATEUSZ PIOTROWSKI is a FreeBSD ports and documentation committer based in Berlin. 

He enjoys troubleshooting bugs, scripting automation, and designing robust software systems 
(always thoroughly documenting everything along the way). Recently, his interests have drifted 
toward tracing and performance engineering. When he is not hacking on the supposedly deter- 
ministic circuitry of modern software, he is exploring the ever-changing dynamics within society 
and culture. 
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Netflix (NASDAQ: NFLX) is the world’s leading streaming entertainment service with 183 million 
paid memberships in over 190 countries enjoying TV series, documentaries and feature films 
across a wide variety of genres and languages. Members can watch as much as they want, 


anytime, anywhere, on any internet-connected screen. Members can 
play, pause and resume watching, all without commercials or commit- 
ments. www.nettlix.com 

Open Connect is the name of the global network that is responsible 
for delivering Netflix TV shows and movies to members world-wide. 
This type of network is typically referred to as a Content Delivery Net- 
work, or CDN, because its job is to deliver internet-based content (via 
HTTP/HTTPS) efficiently by bringing the content that people watch 
close to where they're watching it. Open Connect Appliances run a 
lightly customized version of FreeBSD. https://openconnect.nettlix.com/ 
Open-Connect-Overview.pdf 

Netflix employs several FreeBSD committers and additional mem- 
bers of the Open Connect team also contribute code upstream. 








Open ect Pushes r 100 Tb/s Peak 

Those of us old enough to nee the dot com a telecom 
boom may recall the emblematic 1999 Quest Communications adver- 
tisement in which a weary traveler checks into a hotel in the middle of 
nowhere. The clerk promises a lackluster breakfast, but entertainment? 
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That they have in spades. “Every movie ever made, in any language, anytime day or night.” 
Flabbergasted, the guest wonders aloud “how is that possible?” How indeed! (read on). 
Twenty years later, and hotel TVs are some of the last devices to provide every movie ever 


made. Technology, it seems, is not without a sense of irony. 


No discussion of the latest trends in streaming entertainment and the technology that makes 
it possible is complete without Netflix. As of April 2019, the Netflix U.S. catalog consisted of 
47,000 TV shows and 4,000 movies. Netflix reports that the global Open Connect Network 
pushes over 100 Th/s of traffic at peak. According to Sandvine, this represented about 15% of 





total internet traffic in 2019. 
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Open Connect: A Network And A Program 
Netflix began the Open Connect initiative in 2011 as a response to the ever-increasing scale 
of Netflix streaming. Two primary reasons motivated the program: 


1. As Netflix grew to be a significant portion of overall traffic on consumer Internet Service 
Provider (ISP) networks, it became important to be able to work with those ISPs in a direct 
and collaborative way 

2. Creating a content delivery solution customized for Netflix allowed their engineers to 
design a proactive, directed caching solution that is much more efficient than standard 
demand-driven CDNs. The directed caching architecture reduces the overall demand on 
upstream network capacity by several orders of magnitude. 


Netflix Playback Process 
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Netflix in AWS 


The Network 

Most CDNs work in what's called a demand-driven way. This means that what the network 
caches and where is determined by what is requested in a particular area. For general purpose 
CDNs where there is limited ability to predict the content people will want, this works well. 

Because Netflix controls the end user apps and has detailed information about viewing 
trends, they could achieve significant efficiencies moving to a directed CDN. In the Netflix di- 
rected CDN model, their fleet of Open Connect Appliances (OCAs), described in detail below, 
receive daily catalog updates during what are called Fill windows when viewing is very low. 


The Program 

Netflix has an open peering policy, meaning they will peer with any ISP that agrees with the 
terms of the program. Open peering improves internet user experience by localizing traffic. It also 
has the advantage of reducing transit costs, a benefit to Netflix, ISPs, and the internet as a whole. 

In addition to OCAs in Netflix data centers and installed in Internet Exchange Points (IXPs), 
Netflix provides OCAs free of charge to qualifying ISPs for installation directly in the ISP's net- 
work. This increases localization and reduces upstream traffic even further.’ Interestingly, the 
fact that these OCAs are owned by Netflix, but used by the ISP, raised some licensing consider- 
ations that initially drew the Open Connect engineers to FreeBSD for its permissive license.2 


1 See https://openconnect.netflix.com/Open-Connect-Overview.pdf for program information. 
2 https:/Awww.nginx.com/blog/why-nettlix-chose-nginx-as-the-heart-of-its-cdn/ 








FreeBSD Journal • January/February 2021 | 22 


3 of 7 


FreeBSD CASE STUDY 





Open Connect Appliances 
The workhorses of the Open Connect CDN are the Open Connect Appliances, or OCAs for short. 
These appliances, of which there are three primary configurations, run a lightly customized ver- 
sion of FreeBSD head, or development, branch. That such a large and mission critical network 
would run the fast-moving development branch may at first 
blush seem risky. At the 2019 FOSDEM conference, Jonathan 
Looney, Netflix Engineering Manager on the team responsible —_, ; 
for maintaining the OCA operating system, explained the ratio- Running FreeBSD head lets 
nale of tracking the FreeBSD head branch. us deliver large amounts 
First, Jonathan and his team find FreeBSD code to be gener- Of data to our users very 
ally very stable and high quality. Second, they prefer to quickly efficiently, while maintaining 
find and fix the relatively infrequent and mostly low-impact bugs a high velocity of feature 
they do encounter. Otherwise, Jonathan explains, a development development.” 
team that waits for the long-term, or Stable, branch, may end 
up in what he calls a vicious cycle of infrequent merges, many 
conflicts/regressions, and ultimately slower feature velocity. 
Tracking the head branch helps Netflix add features more 
quickly. They also find that tracking the head branch makes collaborating with others in the de- 
velopment community easier. 


— Jonathan Looney, Netflix 





40Gb/s OCA Storage Appliance with 248TB 
storage (2RU form factor) 

e FreeBSD 

e NGINX 

eBIRD internet routing daemon 











Throughput Efficiency 

Just how efficient are these OCAs? Using FreeBSD and commodity parts, Netflix achieves 90 
Gb/s serving TLS-encrypted connections with ~55% CPU on an Intel 6122 CPU, in 1 RU, with 
96GB RAM, and 16TB of NVMe-attached flash storage. 

Because it’s their intention to upstream as much code as they can, all FreeBSD users bene- 
fit from the many enhancements that help Netflix achieve this kind of performance. Some of 
these contributions include NUMA enhancements, Asynchronous send file, Kernel TLS, Pouf al- 
location enhancements, “Unmapped” mbufs, I/O scheduling, TCP algorithms, and TCP logging 
infrastructure. 

In order to achieve this kind of performance cost-effectively, Netflix engineers realized they 
need to reduce context switching between Kernel and user space as much as possible. Async 
sendfile is one key technique that helps with this. 

The new implementation of the sendfile(2) system call, which is a drop-in replacement for 
the previous one, speeds up TCP data transfers because it avoids copying file data into a buffer 
before it’s sent. The new version of sendfile further speeds up and simplifies large data trans- 
fers by supporting asynchronous |/O. 
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The new sendfile is a product of a development partnership between NGINX and Netflix, 
and was released in tandem with a 2016 Netflix service expansion to nearly 200 countries. 
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Increasing Efficiency and Privacy — Kernel TLS 
To protect the privacy of end users, in 2016 Netflix added Transport Layer Security (TLS). Jan 
Ozer summarized this move well in his Streaming Media article: 


Netflix had long deployed DRM to prevent piracy, and it protects customer data 
during account login and any administration via HTTPS. However, the actual transfer 
of the movie data was not protected, so any information contained in the communi- 
cations between the server and client could be accessed by hackers, or by network 
administrators or ISPs. This information could be used to determine which content 
the viewer was watching, and perhaps other details. 


Adding TLS encryption efficiently required additional performance enhancements to the 
OCA software stack. That's because existing TLS techniques relied on the web server — an ap- 
proach that Nettflix’s Director of Streaming Standards Mark Watson reported in 2014 would di- 
minish capacity “between 30-53%.” 

The answer is kernel-side TLS, or KTLS for short, which marries TLS with the new send- 
file model. This hybrid TLS scheme (described by John Baldwin in this vBSDCon 2019 ses- 
sion) keeps session management in the application space, and inserts the bulk encryption into 
the sendfile data pipeline in the kernel. TLS session negotiation and key exchange messages 
are passed from Nginx to the TLS library, and session state resides in the library's application 
space. Once the TLS session is set up and appropriate keys are generated and exchanged with 
the client, those keys become associated with the communication socket for the client and are 
shared into the kernel. 
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Management 


In their EuroBSD 2019 presentation, Drew Gallatin and Slava Shwartsman show how 
kTLS gives a 50 Gb/s boost to bandwidth performance while reducing CPU% . The next 
frontier in TLS performance improvement is something called NIC TLS, where the encryp- 
tion is done in hardware. As the graph on below shows, this promises to reduce CPU utili- 
zation significantly. 


Netflix Video Serving with TLS 


Kernel TLS Performance 90Gb/s, 68% CPU (SW), 35% CPU (T6 NIC kTLS) 
Original (~2016) Netflix 100G NVME flash appliance 


32 HTT), 128GB DC 2400MT/s, 1x1 


kTLS vs Userspace 
Bandwidth Ecru 


NO KTLS SW KTLS NIC KTLS 
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Getting to 200 Gb/s with NUMA 

With no end in sight to members’ demand for more shows and higher definition, Netflix 
continues to look for ways to increase the throughput of OCAs. With the evolution of high 
core count systems, the team has been developing and testing Non Uniform Memory Architec 
ture, or NUMA, support since 2014, and that is now beginning to show results. Where a typi- 
cal system has a single CPU, disk and memory, a NUMA system can have many more. As with 
sendfile and TLS, this can present throughput-sapping bottlenecks that Netflix engineers have 
been hard at work to minimize. 

NUMA makes it cheaper for a CPU to access local resources (e.g. memory) and more expen- 
sive for it to access resources attached to another node. Consequently, memory and I/O locality 
impacts performance. For Netflix to take advantage of NUMA’‘s greater computation density, 
they had to come up with a way to keep as much of the disk-to-CPU-to-network traffic local 
to a node and minimize performance-sapping NUMA bus transfers. This led to enhancements, 
which are in various stages of being merged upstream, including: 


e Allocating NUMA local memory to back files sent via sendfile(9) 

e Allocating NUMA local memory for Kernel TLS cryptobuffers 

e Directing connections to TCP Pacers and kTLS workers bound to the local domain 

e Directing incoming connections to Nginx workers bound to the local domain via 
modifications to SO_REUSEPORT_LB listen sockets 


In tests, these enhancements have improved Xeon performance from 105Gb/s to 191Gb/s 
While reducing NUMA fabric utilization from 40% to 13%. For AMD EPYC, performance in- 
creased from 68Gb/s to 194Gb/s. 


Four Node Configurations are 
Common on AMD EPYC 
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FreeBSD Gives NETFLIX Three Kinds Of Efficiency: Throughput, 


Development, and Operations 
In response to the FAQ “why FreeBSD?” Jonathan says they came for the license and stayed 
for the efficiency — efficiency that Netflix measures in three ways: 


1. Throughput, or performance, efficiency described in the previous section 
2. Development efficiency 
3. Operational efficiency 


From a development perspective, the ease of working with the FreeBSD community helps 
Netflix upstream their enhancements for ongoing maintenance by the community. They also 
enjoy collaborating with others in the community that are working on the same area. Sharing 
code with these other community members can improve the code all parties are developing. 

Finally, the huge fleet of OCAs requires sophisticated tooling for monitoring and operations. 
Some of the tools they've needed already existed, and the rest they have written. For the latter, 
Jonathan has found FreeBSD does a good job surfacing the necessary hooks and, where not, 
the team has been able to implement them. 


What's Coming Next from the Open Connect Brain Trust 

In addition to NUMA and ongoing exploration of NIC TLS, the team is working on up- 
streaming some enhancements to kTLS and on UFS enhancements. 

In closing, the massive scale of Open Connect combined with the team’s focus on efficien- 
cy and their commitment to open source means that every FreeBSD user with a similar use case 
can reap the same performance benefits. The ability to turn on kTLS and take advantage of 
Async Sendfile allows anyone serving static content over HTTPS to extend their hardware life- 
time, reduce density, and deliver a great user experience more efficiently. 


GREG WALLACE is a freelance technology marketer who has been working with open source 
software and communities since 2005. In addition to his current work with the FreeBSD Foun- 

dation, Greg dabbles in Kubernetes, security, DevOps, and routing. Previously, he led marketing 
for Node.js, ODPi, and Hyperledger. 
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FreeBSD 
for the 


Writing Scholar 


BY COREY J. STEPHAN 


erhaps it is natural that the sort of mind that thrives on piecing together minutia within 

one (not computer-related) academic discipline tends to be different from the sort of mind 

that thrives on learning to use cutting-edge technology. If all that an information technol- 
ogy team provides (or allows) at a workstation is either bloatware from Redmond or spyware 
from Cupertino, then one might (fairly) assume that nothing better exists. Each week, the typical 
academic must balance preparing lessons, lecturing, grading, attending meetings, and holding 
office hours—all while struggling to reserve blocks of time (inevitably too small) for personal re- 
search and writing. The stereotype of the absent-minded scholar often holds true (I write while 
looking in the mirror) not only because of a personal propensity for aloofness but also because 
workdays are disorderly by institutional design. Normally, the Frankenstein Monster-esque com- 
puter setup that | notice while chatting in a windowless, book-filled office is only one piece in a 
particular scholar’s chaotic work life. With so much pressure to produce materials for publication, 
who has time to build a better computer workflow? 

| think that all the traits that a scholar needs in a desktop operating system fit within three 
broad categories: documentation, stability, and security. 

Scholars like documentation atop documentation (atop documentation). If scholars cannot ver- 
ify something for ourselves, then we are unlikely to trust it. Poorly documented operating systems 
cannot withstand the skepticism that is standard in the academy. Organization and documenta- 
tion go hand-in-hand. An operating system whose developers prioritize consistency is probably 
going to be intelligible for the person who takes the time to learn a bit about how it works. Good 
man pages, a clean handbook, and a wiki that a dedicated userbase actively maintains—this tri- 
fecta is the minimum of order that a scholar’s OS must have. 

Scholars do not need the same kind of stability for our workstations as, say, freebsd.org needs 
for its servers, but we do need stability. Twice, my Ryzen-based desktop computer with 16GB 
of RAM has crashed because | was using 13GB in one program to manipulate a manuscript fac 
simile. On another occasion, | almost entered cardiac arrest because an update for LibreOffice’s 
Fresh branch rendered all of my work-in-progress dissertation chapters un-editable until | paused 
and realized that | simply needed to downgrade to LibreOffice Still. To say the least, neither mo- 
ment was pleasant. Scholars work with many long, complicated documents and databases, and 
our success depends on as few of those failing during crunch time as possible. If tools in scholars’ 
toolboxes are not broken, we will not wish for anyone to try to fix them. 
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Security, like stability, means something else for the scholar’s workstation than it does for 
the kinds of environments that a system administrator or software developer has in mind. For 
the scholar, security means privacy. The best scholars share their work with others for feedback 
and expect others to do the same with them. However, academics are also the sort of people 
to be quite selective about those who are or are not able to see what they are doing. Many ac 
ademics work with confidential materials. Any operating system that reports what its end users 
are doing to someone somewhere else or that can be hacked or monitored easily is not suit- 
able for scholarly work. 

FreeBSD is not the only major open-source operating system with superb documentation, sta- 
bility, and security (we must never overlook our siblings in the other *BSDs or cousins working on 
Debian and other great GNU/Linux distributions), but it is undoubtedly one of the best. Informa- 
tion technology professionals often are the only people writing about FreeBSD. They might not re- 
alize that many of the things that make FreeBSD fantastic for servers and embedded systems also 
make it outstanding as a desktop operating system for the writ- 
ing scholar. . 

The documentation and organization of FreeBSD are splen- All the traits that 
did. As a scholar particularly ever in search of historical doc a scholar needs in 
umentation, trying to find order amidst a diversity of ancient 
voices, | find the internal coherency of FreeBSD to be downright a desktop operating 
refreshing. Even kernel building in FreeBSD mainly involves the 


use of human-readable, plain text configuration files. Things system fit within 
make sense because the people creating them aim for them to three broad categories: 
make sense. 

Few people are going to question FreeBSD’s stability. When documentation, 
the FreeBSD Wiki lists certain hardware as being well supported, “~ f 
FreeBSD most likely will not be the cause of a system crash for stability, and security. 


anyone doing (even esoteric) desktop work on that hardware. 

My system crashes were in a different operating system. | doubt 

the same would have happened if | had been using a properly configured FreeBSD-STABLE sys- 
tem and the long-term service branches of my main third-party software applications. 

Finally, FreeBSD is fully open and markedly secure. With FreeBSD, scholars can trust that our in- 
tellectual property is not monitored by a suit at some hypothetical FreeBSD headquarters in Silicon 
Valley. FreeBSD is sensibly secure by default and sensible enough to secure. 

| proudly recommend FreeBSD for fellow writing scholars. Install FreeBSD. Install the X Win- 
dow System. Install a dynamic tiling window manager with plain text configuration like FreeBSD 
consistently uses, such as spectrwm or i3wm, or else a simple traditional desktop environment 
(XFCE is trustworthy). Install LibreOffice Still and a decent citation management tool that in- 
tegrates with LibreOffice and handles materials from your discipline (Zotero runs flawlessly in 
Wine, and FreeBSD has its own native options). Install discipline-specific command-line interface 
tools. The pkg system makes a good deal of third-party desktop software simple to install. En- 
joy researching and writing with a logical, stable, and overall solid operating system. Your pub- 
lishing schedule no longer will be subject to setbacks from your software. 


COREY STEPHAN is a Ph.D. candidate in the Department of Theology at Marquette University 
in Milwaukee, Wisconsin. He proudly makes exclusive use of F.0.S.S. tools to assist his research 
in the history of Christian theology. His professional website is coreystephan.com. 
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On Top of the World 


BY BENEDICT REUSCHLING 


This column covers ports and packages for FreeBSD that are useful 
in some way, peculiar, or otherwise good to know about. Ports 
extend the base OS functionality and make sure you get something 
done or, simply, put a smile on your face. Come along for the ride, 
maybe you'll find something new. 


ver since | began working with Unix, | remember using top(1). For those of you who 

have been living under a Unix rock, you should really climb on top. With its continuing 

display of processes, it gives you an instant view of what Is going on in your machine— 

process wise. Compared to the static output of ps(1), it decorates at least one corner of 
any serious Unix sysadmin screen to make it look busy. Fancy screensaver or not, a quick glance 
can help in an early evaluation of system load. Of course, the BSDs are no different and they 
even provide some extra features that other top implementations don’t have. For I/O, 


gives you reasons every second to finally replace that hard disk with something, well, flashier. 
If you want to just list the ten most active processes that hog your system? Simply type 


into your terminal to get it. Very intuitive! Also, exiting out of top is easier for beginners than 
getting out of vi (hint: q is involved). Perhaps that is one of the reasons this program has been 
cloned and rewritten for other purposes over the years. Whenever there is a need to figure out 
why things are sluggish, top is one way to figure it out. Users tend to complain about it only to 
discover that their own processes are grinding the system to a near standstill. 

The most popular rewrite is probably sysutils/htop, which extends the base top with col- 
ors and a customizable display. From adding a remaining battery display on your laptop to a 
display of the hostname for the ssh system-to-system hoppers among us, it all can be config- 
ured. Process views offer a neat tree view of threads spawned by your shell or system daemons 
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processing data. My years of playing DOS games and hitting the spacebar all too often has me 
confused in htop, though. On the FreeBSD top implementation, this causes the display to re- 
fresh. And on htop, it selects the process under the cursor. | should study the man page to see 
what it does instead of relying on top being the same everywhere. 

Speaking of old DOS games: after installing sysutils/bashtop and invoking it for the first 
time, | can’t shake the feeling that I’m running in professional protected mode runtime again. 
It freezes completely and the homepage redirects me to bpytop instead. Just looking at the 
github page [https://github.com/aristocratos/bpytop], | see it not only lists FreeBSD support, but 
a whole lot of others, too. My fingers quickly type 


into my terminal to pull down the latest ported version. The application runs and l'm stunned 
for a second: another initialization routine. My trained gaming fingers are quick enough to cap- 
ture it in a screenshot for readers before the main display appears. You may think I’m taking 
this to the top, but what follows is as close to a drug trip that | have probably ever come to 

in my life—the colors are just too much for me. Nevertheless, it shows a lot of good informa- 
tion about your system. The little dots in the top left corner give me a combined view of each 
of the 24 CPUs | have on this system. Selecting a process and hitting enter yields more details 
about it. Network and memory are also shown on the same screen. Certainly neat, but now | 
need to get my eyeballs replaced—| think | still have some in the fridge... 


Version: 1.0.59 


Loading theme and creating colors... 


Doing some maths and drawing... 

Setting up signal handlers... 

Starting input reader thread... 

Starting data collection and drawer thread... 
Collecting data and drawing... 





OK, Dilbert’s Topper would say “That’s nothing...” so l'm searching freshports.org look- 
ing for other top-like utilities. Sure enough, nearly every letter of the Alphabet seems to have 
been put in front of top in one way or the other. From sysutils/atop that works quite well 
on FreeBSD despite the man page claiming it is a resource monitor for Linux, databases/mtop 
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or mytop to view your MySQL processes, to pgtop (same for Postgres) there is plenty to select 
from. The only thing I’m missing is stop—stop a process. What happened to “programs should 
do one thing and do it well?” Maybe l'm old fashioned in this regard! 

Network admins will probably take a look at dns/dnstop to capture and peek into DNS 
traffic flowing by. Simpler views than bpytop for 
sure but it has all one needs. Or try out net-mgmt/ 
bandwhich to pinpoint where your bandwidth is 





going all day. 

One thing | found that has not been ported What happened to 
yet and has nothing to do with a process view- “Prog rams should do 
er is topgrade [https://github.com/r-darwish/top- 
grade]. It upgrades not only from one, but all one thing and. do it 
package managers you have on your system. Per- x, P 
haps playing the Top Gun theme while doing it, well? Maybe I'm-old 


the logo certainly brings that to mind. The server | 
was running this on has no sound chip, so | can’t 

tell. Imagine all the datacenter workers scrambling 
to figure out which machine this comes from in a 
rack of machines! 

| also remember (off the top of my head, of course) other top-like programs not yet ported 
to FreeBSD. The movie TRON seems to have inspired eDEX-UI [httos://github.com/GitSquared/ 
edex-ui] with its futuristic design. Can we get that one ported, pretty please (with sugar on 
top)? 

There is a base RUST library called tui-rs [https://github.com/fdehau/tui-rs] which provides 
the building blocks for many other flexible and dynamic window-like displays in the terminal. I'll 
mention just one of them here which seems to be what tail(1) is to top’s head(1): bottom 
[https://github.com/ClementIsang/bottom]. While there, sysutils/gotop may interest you 
with its squiggly line display of CPU usage. On laptops and servers, it tries to determine CPU 
temperatures as well. 

| would refer you to www.unixtop.org to get some history about this utility if it were not 
down. Luckily, archive.org has a copy from 2017 that can be used. Wikipedia also has enlight- 
ening information, so I'll leave you to that. Here's hoping this column wasn’t too much over the 
top and you could add some utilities to your Unix toolbox. 


fashioned in this regard! 








BENEDICT REUSCHLING is a documentation committer in the FreeBSD project and member 
of the documentation engineering team. He serves on the board of directors of the FreeBSD 
Foundation as vice president. In the past, he served on the FreeBSD core team for two terms. 
He administers a big data cluster at the University of Applied Sciences, Darmstadt, Germa- 

ny. He’s also teaching a course “Unix for Developers” for undergraduates. Together with Allan 
Jude, he is host of the weekly bsdnow.tv podcast. 
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FreeBSD 


® 


Donate to the Foundation! 


You already know that FreeBSD is an internationally 
recognized leader in providing a high-performance, 
secure, and stable operating system. It's because of 
you. Your donations have a direct impact on the Project. 


Please consider making a gift to support FreeBSD for the 
coming year. It’s only with your help that we can continue 
and increase our support to make FreeBSD the high- 

performance, secure, and reliable OS you know and love! 


Your investment will help: 
e Funding Projects to Advance FreeBSD 


e Increasing Our FreeBSD Advocacy and 
Marketing Efforts 


e Providing Additional Conference 
Resources and Travel Grants 


* Continued Development of the FreeBSD 
Journal 


e Protecting FreeBSD IP and Providing 
Legal Support to the Project 


e Purchasing Hardware to Build and 
Improve FreeBSD Project Infrastructure 


Making a donation is quick and easy. 
freebsdfoundation.org/donate 
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BSD Events taking place through March 2021 


BY ANNE DICKISON 

















Please send details of any FreeBSD related events or events that are of interest for FreeBSD 
users which are not listed here to freebsd-doc@FreeBSD.org. 





Users with organizational software that uses the iCalendar format can subscribe to the FreeBSD 
events calendar which contains all of the events listed here. 


CSC APRICOT APRICOT 2021 


Asia Pacific Regional March 1-4, 2021 
oS operational Technologies. VIRTUAL 
Representing Asia Pacific's largest international Internet conference, Asia Pacific Regional Internet 
Conference on Operational Technologies (APRICOT) draws many of the world’s best Internet 
engineers, operators, researchers, service providers, users and policy communities from over 50 
countries to teach, present, and do their own human networking. The ten-day summit consists 
of seminars, workshops, tutorials, conference sessions, birds-of-a-feather (BOFs), and other 
forums with the goal of spreading and sharing the knowledge required to operate the Internet 
within the Asia Pacific region. This year, the conference will happen virtually. 



































FOSS FOSSASIA 


March 13-21, 2021 


aSa vru 


Join the FreeBSD Foundation at the FOSSASIA Virtual Summit 2021. Taking place March 13-21, 
2021, FOSSASIA provides the opportunity to learn about open technologies and to connect with 
lead developers and technologists from Free and Open Source software (FOSS) and Open Hardware 
projects from around the world. Registration is free of charge. 











FreeBSD Fridays 

https://freebsdfoundation.org/freebsd-fridays/ 

Our next FreeBSD Fridays session will be March 26. 

Past FreeBSD Fridays sessions are available at: https://freebsdfoundation.org/freebsd-fridays/ 








FreeBSD Office Hours 

https://wiki.freebsd.org/OfficeHours 

Join members of the FreeBSD community for FreeBSD Office Hours. From general Q&A to 
topicbased demos and tutorials, Office Hours is a great way to get answers to your FreeBSD- 
related questions. There will be a FreeBSD Core Team Office Hours session on March 17. 





Past episodes can be found at the FreeBSD YouTube Channel 
https:/Awww.youtube.com/c/FreeBSDProject. 
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